classdef standard_bootstrap_CV

% standard_bootstrap_CV is an object that implements the main functionality of
% a cross-validator (CV) object.  Namely, it takes a classifier (CL), a datasource (DS)
% and optionally feature preprocessor (FP) objects, and it run a cross-validation
% decoding scheme by training and testing the classifier with data generated
% from the datasource object (and possibly applying feature pre-processing first).
% The cross-validation procedure is run multiple times
% in a bootstrap-like manner in which the datasource generates different training
% and test splits of the data, and the final decoding performance can be averaged over these
% bootstrap runs.  
%
% The run_cv_decoding method for this object returns a "DECODING_RESULTS" structure
% that contains zero-one loss results as well as potentially 3 other types of results
% (normalized rank results, average decision values, and area under ROC curve results).
% Confusion matrices and different measures of the variability of the decoding results
% can also be returned.  The options that allow one to change the behavior of this
% object and the results that this object returns are described in more detail below.
%
% The constructor for this object has the form:  
%  cv = basic_bootstrap_CV(the_datasource, the_classifier, the_feature_preprocessors)
%  where the_datasource is a DS object, the_classifier is a CL object, and 
%  the_feature_preprocessors is an optional cell array that contains FP objects.  
%
%
% Properties that can be set are:
%
%   num_bootstrap_runs (default = 50), which is the minimum number of bootstrap runs
%      that the decoding procedure will be run.
%
%   test_only_at_training_times (default = 0).  If this is set to zero than the classifier
%      will be trained and testing at all times, which will create a matrix of results
%      that have the decoding accuracies for training at time 1 and testing at time 2.  
%      This allows one to test if the neural code is stationary (see Meyers et al, 2008).
%      If this properties is set to 1, then the results will only be for training and testing
%      at the same time (which can cause the run_cv_decoding method to potentially run 
%      much faster, particularly if it classifier is slow at returning test results.
%        
%
% If a classifier is used that returns decision values along with zero-one loss results,
%  then the additional decoding accuracy results can be calculated and returned by setting
%  the following options:
%
%   save_results    
%       .normalized_rank (default = 1), returns normalized rank results
%   	.decision_values (default = 1), returns the decision values
%       .extended_decision_values (default is 0). If this property is set to 1 
%           then all the decision values for the correct class are saved. If 
%           this property is set to 2, then all decision values from all classes are
%           saved (doing this can potentially take up a lot of memory/disk space).
%       .ROC_AUC (default = 1), returns area under ROC curve results.  These results 
%           are calculated separately for each class k, with the test decision values from 
%           class k being the positive examples, and all other decision values being the negative
%           examples.  The results can be calculated separately on the test points on each CV split, 
%           (save_results.ROC_AUC = 3) or from combining all decision values over all cross-validation splits
%           (save_results.ROC_AUC = 2).  If save_results.ROC_AUC = 1, then both separate CV results, and combined
%           CV results will be saved.  
%
%  A number of parameters can be set to create confusion matrices for the results.
%   A confusion matrix is a matrix in which the columns express the true class label
%   and the rows express the predicted label.  Thus, for example, if column 2, and row 3
%   had a value of 7, it would mean that there were 7 times in which class 2 was mistakenly
%   predicted to be class 3 (summing the columns will give the total number of examples for 
%   each class, which can be used to turn the confusion matrix into a misclassification probability
%   distribution).  The following parameters can be set to create confusion matrices
%
%   confunsion_matrix_params
%       .create_confusion_matrix (defatul is 1).  Create a basic zero-one loss confusion matrix
%       .save_confusion_matrix_only_train_and_test_at_same_time (default is 1). Saves the 
%           confusion matrix only for when training and testing was done at the same time.
%       .create_all_test_points_separate_confusion_matrix (default is 0).  Creates a confusion matrix 
%           where all test points are separate, i.e., if there are two test points from the same
%           class on a given CV split, these two test points will be given separate column
%           in the confusion matrix (this is useful when ID are remapped using the 
%           specify_training_and_test_label_remapping_DS datasource).
%
%                    
%  The display_progress options allow one to display the decoding result 
%   (for different result types) as the decoding procedure as it is running.
%   The results displayed show the mean decoding results, as well as as a measure
%   of the variability of the results (which gives a sense of whether enough bootstrap
%   iterations have been run so that the results have converged to a stable solution).  The measure of
%   variability of the results is calculated by computing the mean decoding results 
%   if the i-th bootstrap run was left out (i.e., if there was one less bootstrap run).  This
%   is done for each bootstrap run, and the standard deviation is taken over these 
%   one-bootstrap-left-out means.  Thus this measure gives a sense of how much the results
%   would vary if one less bootstrap iteration was run (one can get a rough sense of whether
%   the results have converged if one adds these to the mean results - overall these numbers
%   should be very small).  The following options allow one to display progress of different
%   result types (the results for different types are displayed for all times, and are separated
%   by NaNs, i.e,. mean_type1 stdev_type1 NaN mean_type2 stdev_type2, NaN, etc.).
%
%   display_progress
%       .zero_one_loss (default = 1).  Display zero-one loss results         
%       .normalized_rank (default = 1). Display noramlized rank results
%       .decision_values (default = 0). Display decision values
%       .separate_CV_ROC_results (default = 0).  Display ROC AUC results computed separately on each CV split
%       .combined_CV_ROC_results (default = 0). Display ROC AUC results combined points from all CV splits
%       .training_time_to_display_results (default = -1).  The training time for which to display the test result values. 
%          A value of -1 means that the displayed results will for training at the test time points 
%          (or if only 1 training time was used, then a value of -1 means use that time for all test times). 
%
%
%
%  The run_cv_decoding method returns a structure called DECODING_RESULTS that contains
%   the following properties.
%
% DECODING_RESULTS
%
%   .FP_INFO{iFP}   If any of the feature preprosessing algorithms had returned information to be
%       saved via their get_current_FP_info_to_save method, then this structure will contain 
%       the additional information that the FP algorithm wanted to be saved.  
%
%   .BOOTSTRAP_CV_PARAMETERS  This structure contains additional information about parameters
%       used in the decoding procedure.  This structure has the following fields:
%
%       .num_test_points_on_each_CV_run      The number of test points on each CV split
%       .num_training_points_on_each_CV_run  The number of training points on each CV split
%       .dimension_of_data_points            The dimensionality of the training/test points
%       .unique_labels                       A vector containing the unique test labels that were used
%       .num_unique_labels                   The number of unique labels (classes) 
%       .num_CV_runs                         The number of cross-validation splits of the data
%       .toolbox_version_number              The number of current neural decoding toolbox being used
%
%   .ZERO_ONE_LOSS_RESULTS  This structure contains the main (zero-one loss) decoding results.
%       The following results can be returned.
%
%       .decoding_results [num_training_times x num_test_times x num_CV_splits x num_bootstrap_runs]
%           This tensor contains all the decoding results separate for each bootstrap run and
%           cross-validation split.
%       .mean_decoding_results: [num_training_times x num_test_times]  This contains the mean
%           decoding results averaged over all bootstrap runs and CV splits
%       .confusion_matrix_results  If the confusion matrix field options are set (e.g., 
%           cv.confusion_matrix_params.create_confusion_matrix == 1), then this structure
%           can contain the following fields:
%           .confusion_matrix [num_predicted_classes x num_actual_classes x num_training_times x num_test_times]
%               This contains the confusion matrix specifying how often a test point j was classified as 
%               belonging to class i (where j indicates column indecies, and i indicates row indecies).  If 
%               .save_confusion_matrix_only_train_and_test_at_same_time = 1, then the confusion matrix is
%               [num_predicted_classes x num_actual_classes x num_training_times] large and only contains the confusion
%               matrices when training and testing at the same time period (which saves a lot of disk space when saving the results).
%           .label_remapping  If the labels given to the classifier are not consecutive integers, this 
%               vector indicates how the labels have been remapped on to the columns of the confusion matrix
%               (i.e., the first value in the vector indicates the class in the first row of the confusion matrix, etc.).
%           .all_test_points_separate_confusion_matrix If the create_all_test_points_separate_confusion_matrix flag is set to 1, 
%               then this variable contains a confusion matrix that is [num_test_points x num_actual_classes x num_training_times x num_test_times]
%               large, with each test point in a given CV split being given a separate row in the confusion matrix (this confusion matrix will 
%               only differ from the regular confusion matrix if there are multiple test points from the same class in
%               a given CV split).  This matrix can be useful if the labels have been remapped to different classes using
%               the specify_training_and_test_label_remapping_DS datasource in order to see how the confusion in the original unremapped labels
%       .stdev  This structure contains information about the variability of the results.  This structure has the following fields:
%           .all_single_CV_vals(num_boot, num_CV, num_train, num_test)
%               Each CV run produces a value for each test point (i.e., 0 and 1's, 
%               normalized ranks, or decision values).  This matrix contains the stdevs
%               over these values for each CV run.
%           .all_single_CV_vals_combined(num_boot, num_train, num_test)
%               This is the same as .all_single_CV_vals but all the values from the
%               CV runs are combined together first before
%               the stdev is taken (this is done separately for each bootstrap run).  
%           .over_CVs(num_boot), num_train, num_test)
%               The mean decoding results in each CV run are computed (i.e., the mean
%               of the 0, 1's, ranks or decision values of the points in a CV split)
%               and the stdev over these mean CVs are calculated (this is done separately
%               for each bootstrap run).
%           .over_CVs_combined_over_bootstraps(num_train, num_test)
%               This is the same as .over_CVs but all the values from all the bootstrap 
%               runs are combined before the stdev is taken.
%           .over_boostraps(num_train, num_test)
%               This calculates the mean over all the (mean) CV values for a bootstrap run,
%               and then calculates the stdev over the different bootstrap runs.  
%
%         It should be noted that .over_CVs, .over_CVs_combined_over_bootstraps and
%           .over_boostraps could have all been computed running the decoding experiment
%           using the values in .decoding_results, but we precompute them here for convenience.
%
%
%    For classifiers that return decision values, this CV object can return additional results in 
%     the structures NORMALIZED_RANK_RESULTS, DECISION_VALUES and ROC_AUC_RESULTS that are described below.  
%
%    .NORMALIZED_RANK_RESULTS  This structure contains the normalized rank results.  
%       Normalized rank results are the results based on using the prediction values from
%       the classifier to create an ordered list of predictions (i.e., the most likely class is x, 
%       second most likely class is y, etc.), and then assessing how far done on the list is the
%       corect label.  The results are normalized so that perfect prediction has a value of 1, 
%       chance has a value of .5 and having the last prediction be the correct one leads to a value of 0.
%       The .NORMALIZED_RANK_RESULTS has the same .mean_decoding_results and .stdev results 
%       as .ZERO_ONE_LOSS_RESULTS but the different confusion matrix values.  The confusion matrix
%       for the normalized rank results is in the field .confusion_matrix_results.rank_confusion_matrix
%       and contains matrix that is [num_predicted_classes x num_actual_classes x num_training_times x num_test_times]
%       which contains values in the ith row and jth column for the average normalized rank ith predicted class when test points
%       from the jth actual class were presented (i.e., how high up on the predicted labels list is the ith class 
%       when test points from the jth class are shown).  There is also a field .confusion_matrix_results.rank_confusion_matrix_label_mapping 
%       that contains the labels that correspond to the columns of the rank confusion matrix.
%
%    .DECISION_VALUES  This structure contains the decision values.  It has all the same fields as the ZERO_ONE_LOSS_RESULTS
%       except that there is no confusion matrix for this result type.  If save_results.extended_decision_values = 1, then
%       a matrix .classifier_decision_values of dimension [num_train_times x num_test_times x num_cv_splits x num_bootstrap_runs x num_test_points]
%       will be returned that will have the decision values for the correct/actual class for each test point.  
%       If save_results.extended_decision_values = 2, then a matrix .all_classifier_decision_values is returned that has dimensions
%       [num_train_times x num_test_times x num_cv_splits x num_bootstrap_runs x num_test_points x num_classes]
%       that contains all the decision values for every class (not just the correct class).  Thus this result contains 
%       all the information from the whole decoding process (and consequentially it could take up a lot of diskspace/memory).
%       A matrix .all_classifier_decision_labels [num_train_times x num_test_times x  num_cv_splits x num_bootstrap_runs x num_test_points] 
%       will also be returned that contains all the labels that were used.  From these two structures it is possible to derive all 
%       other decoding measures that are returned.  
%
%    .ROC_AUC_RESULTS  This structure contains results that measure the area under an receiver operator characteristic (ROC) curves
%       that are created separately for each class from the decision values.  ROC curves graph the proportion of positive test 
%       points correctly classified (true positive rate) as a function of the proportion of negative test points incorrectly
%       classified (false positive rate).  The area under this curve (ROC AUC) gives a measure of decoding accuracy that has a number of
%       useful properties, including the fact that it is invariant to the ratio of positive to negative examples, and that
%       it can be used to determine decoding accuracies when multiple correct classes are present at the same time.  The results
%       in this structure have the following fields:
%   
%       .separate_CV_ROC_results  This structure calculate the ROC AUC separately for each CV split of the data.  The
%           advantage of calculating this separately for each CV split is that it maintains the independence of the
%           decoding results across CV splits.  The disadvantage is that main times there will only be one or a couple of
%           test points for each class which will lead to a highly variable estimate of the ROC AUC.  For this reason it is often
%           better to use the .combined_CV_ROC_results results described below.
%
%       .combined_CV_ROC_results  This structure calculate the ROC AUC by combining all the decision values from all 
%           the test points across the different CV splits.  While the test points should all be independent from one another
%           the classifiers used to evaluate these test points are highly related (since they are most likely trained using very similar
%           data), thus these results could be slightly biased.  However, since a much larger number of points are used to create
%           these ROC curves, the results are more likely to be more sensitive (i.e., these results are slighly more likely to contain
%           type 1 errors but much less likely to create type 2 errors compared to using the .separate_CV_ROC_result structure).
%
%         Both the .separate_CV_ROC_results .combined_CV_ROC_results have the following fields:
%        
%             .decoding_results this contains a  [num_training_times x num_test_times x num_bootstrap_runs x (num_CV_splits) x num_classes] 
%                matrix that contains the ROC AUC results (for the separate_CV_ROC_results there are 5 'dimensions' while for the
%                combined_CV_ROC_results results there are only 4 dimensions since the results combine data from the different CV splits
%                when calculating the ROC AUC results).
%             .mean_decoding_results  This is a [num_training_times x num_test_times] sized matrix that contains the mean ROC AUC
%                values averaged over bootstrap runs the different classes and for the separate_CV_ROC_results, the results are
%                also averaged over CV splits.  
%             .stdev  This field contains the following measure of variability of the ROC AUC results:
%                .over_classes(num_train, num_test, num_boot, [num_CV])  Computes the stdev over the results for each class.
%                .over_bootstraps(num_train, num_test)  Takes the mean over all the classes ([and CV runs]), and computes
%                      the stdev over the bootstrap runs.
%                Additionally, .separate_CV_results has the following measure:
%                  .over_CVs(num_train, num_test, num_boot, num_classes)  Takes the mean over classes and computer the variability over the CV runs.
% 




%==========================================================================

%     This code is part of the Neural Decoding Toolbox.
%     Copyright (C) 2011 by Ethan Meyers (emeyers@mit.edu)
% 
%     This program is free software: you can redistribute it and/or modify
%     it under the terms of the GNU General Public License as published by
%     the Free Software Foundation, either version 3 of the License, or
%     (at your option) any later version.
% 
%     This program is distributed in the hope that it will be useful,
%     but WITHOUT ANY WARRANTY; without even the implied warranty of
%     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
%     GNU General Public License for more details.
% 
%     You should have received a copy of the GNU General Public License
%     along with this program.  If not, see <http://www.gnu.org/licenses/>.
    
%==========================================================================   




    properties 

        datasource;
        classifier;
        feature_preprocessors = [];   % cell array of FP objects
       
        num_bootstrap_runs = 50;
               
        test_only_at_training_times = 0;
        
        
        % result types to save
        save_results = struct('ROC_AUC', 1, 'normalized_rank', 1, 'decision_values', 1, 'extended_decision_values', 0);
                              
        % confusion matrix variables
        confusion_matrix_params = struct('create_confusion_matrix', 1, 'create_all_test_points_separate_confusion_matrix', 0, 'save_confusion_matrix_only_train_and_test_at_same_time', 1);
        
        % displaying bootstrap progress is not possible when using the Matlab's parallel toolbox
        % so the following variables have been commented out.
        % flags for displaying the progress of the decoding procedure     
        display_progress = struct('bootstrap_run_time', 1, 'zero_one_loss', 1, 'normalized_rank', 1, 'decision_values', 0, 'separate_CV_ROC_results', 0, ...
            'combined_CV_ROC_results', 0, 'training_time_to_display_results', -1);    % display how long each bootstarp iteration took
                   
        % could potentially convert the display_progress results itno criteria as to when enough bootstrap iterations have been run...
        
    end
    
  
    methods
        
        
        % the constructor  - pass the main objects needed to run the cv decoding
        function cv = standard_bootstrap_CV(the_datasource, the_classifier, the_feature_preprocessors)
                            
            cv.datasource = the_datasource;
            cv.classifier =  the_classifier;
        
            
            if nargin > 2    % don't need to specify preprocessing algorithms to run the decoding analysis
                cv.feature_preprocessors = the_feature_preprocessors;   
            end
                      
        end
        
        
        
        
       
        function DECODING_RESULTS = run_cv_decoding(cv) 

            
            % sanity checks
            if (cv.confusion_matrix_params.save_confusion_matrix_only_train_and_test_at_same_time == 0) && (cv.test_only_at_training_times == 1)
                error(['To create a confusion matrix that has all training and times one has to run a decoding procedure at all training and test times. '... 
                    'Thus one can not have cv.confusion_matrix_params.save_confusion_matrix_only_train_and_test_at_same_time = 0 and test_only_at_training_times = 1']);
            end
            
            
            
            if isempty(cv.test_only_at_training_times)
                cv.test_only_at_training_times = 0;
            end
            

            datasource = cv.datasource;
            %FP_info_to_save = [];
            
            parfor iBootstrap = 1:cv.num_bootstrap_runs   % parfor

                if  cv.display_progress.bootstrap_run_time 
                    tic
                    iBootstrap
                end

                % needed to add these variables here to get parfor working (matlab is lame)
                curr_feature_preprocessors = cv.feature_preprocessors;
                classifier = cv.classifier;   % might be faster if I copy the classifier object since calls to object fields are slow   
                curr_bootstrap_DECODING_RESULTS = [];
                all_CV_decision_vals_label_present = []; 
                all_CV_decision_vals_label_absent = []; 
                curr_bootstrap_rank_confusion_matrix = [];
                save_normalized_rank = cv.save_results.normalized_rank;
                save_decision_values = cv.save_results.decision_values;
                save_roc_auc = cv.save_results.ROC_AUC;
                test_only_at_training_times = cv.test_only_at_training_times;

                curr_results_to_display = []; 
                
                
                [all_XTr all_YTr all_XTe all_YTe] = datasource.get_data;
                %[all_XTr all_YTr all_XTe all_YTe ADDITIONAL_DATASOURCE_INFO] = get_data(datasource);   % additional info is useful in specify_training_test_datasource to get the original labels (before remapped) - these can be used for some feature preprocessing algorithms



                % pre-allocating memory (is this necessary, and if so, even when not saving rank or decision values)
                if test_only_at_training_times == 1
                    curr_total_CV_zero_one_loss_results = cell(size(all_XTr, 2));
                    curr_total_CV_normalized_rank_results = cell(size(all_XTr, 2));
                    curr_total_CV_decision_values_results = cell(size(all_XTr, 2));                   
                else
                    curr_total_CV_zero_one_loss_results = cell(size(all_XTr, 2), size(all_XTe, 2));
                    curr_total_CV_normalized_rank_results = cell(size(all_XTr, 2), size(all_XTe, 2));
                    curr_total_CV_decision_values_results = cell(size(all_XTr, 2), size(all_XTe, 2));
                end
                
                
                 % run through all the CV trials first
                for iCV = 1:size(all_XTr{1}, 2)


                    for iTrainingInterval = 1:size(all_XTr, 2)


                            XTr = all_XTr{iTrainingInterval}{iCV};
                            YTr = all_YTr{iCV};

                            % apply preprocessing to the training data ... 
                            if ~isempty(cv.feature_preprocessors)
                                for iFP = 1:length(cv.feature_preprocessors)

                                    
                                    [curr_feature_preprocessors{iFP} XTr] = curr_feature_preprocessors{iFP}.set_FP_properties_and_get_preprocessed_XTr(XTr, YTr);  % save FP parameters and get normalized XTr

                                    curr_FP_info_to_save = curr_feature_preprocessors{iFP}.get_current_FP_info_to_save;
                                    
                                    % if specified by the user, save preprocessing information 
                                    if ~isempty(curr_FP_info_to_save)

                                        curr_bootstrap_DECODING_RESULTS.FP_INFO{iFP}(iTrainingInterval, iCV) = curr_FP_info_to_save;
                                        
                                        % Since eval statements are not allowed when using Matlab's parallel toobox, the below section has been commented out
                                        %  and the conversion to the proper structures to save FP data is done after the whole bootstrap procedure is complete
                                        %  (this is potentially less efficient in terms of memory/computation time, but it is the only way to get it to work).
                                        %
                                        % field_names = fields(curr_feature_preprocessors{iFP}.preprocessing_information_to_save);  
                                        % for iFieldName = 1:length(field_names)
                                        %     
                                        %     eval(['curr_FP_data_to_save = curr_feature_preprocessors{iFP}.preprocessing_information_to_save.' field_names{iFieldName} ';']); 
                                        %
                                        %     if isnumeric(curr_FP_data_to_save)
                                        %         eval(['DECODING_RESULTS.FP_INFO{iFP}.'  field_names{iFieldName} '(iBootstrap, iCV, iTrainingInterval, :) = curr_FP_data_to_save;']);    % will have to modify this slightly if returning a matrix, etc..
                                        %     else
                                        %         eval(['DECODING_RESULTS.FP_INFO{iFP}.'  field_names{iFieldName} '{iBootstrap, iCV, iTrainingInterval} = curr_FP_data_to_save;']); 
                                        %     end
                                        % end
 
                                     end

                                    
                                end
                            end   % end preprocessing training code
                                                     
                            
                            
                            
                            classifier = classifier.train(XTr, YTr);   % train the classifier 
                            
                            
                            % if one wants to only test at the same time points that were used for training (to speed things up)
                            if test_only_at_training_times == 1
                                test_interval = iTrainingInterval;
                            else
                                test_interval = 1:size(all_XTe, 2);   % otherwise create full train at t1 test at t2 matrix
                            end

                            
                            
                            for iTestInterval = test_interval    
                                

                                XTe = all_XTe{iTestInterval}{iCV};
                                YTe = all_YTe{iCV};


                                % apply feature preprocessing to the test data
                                if ~isempty(cv.feature_preprocessors)
                                    for iFP = 1:length(cv.feature_preprocessors)       
                                        XTe = curr_feature_preprocessors{iFP}.preprocess_data(XTe);                     
                                    end
                                end


                                
                                % test the classifier
                                [predicted_labels decision_values] = classifier.test(XTe);



                                % if creating a confusion matrix, store the appropriate info
                                if cv.confusion_matrix_params.create_confusion_matrix  
                                    if test_only_at_training_times == 1
                                        all_predicted_labels{iBootstrap}(iCV, :, iTrainingInterval, iTestInterval) = predicted_labels;
                                    else
                                        all_predicted_labels{iBootstrap}(iCV, :, iTrainingInterval) = predicted_labels;
                                    end
                                end

  
                                % save the zero-one loss results
                                if test_only_at_training_times == 1
                                    curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.decoding_results(iCV, iTrainingInterval) = ((length(find(predicted_labels - YTe == 0))/length(predicted_labels)));  
                                else
                                    curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.decoding_results(iCV, iTrainingInterval, iTestInterval) = ((length(find(predicted_labels - YTe == 0))/length(predicted_labels)));  
                                end
                            
                                
                                % storing stdevs of single 0-1 classifier results
                                if test_only_at_training_times == 1
                                    curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.all_single_CV_vals(iCV, iTrainingInterval) = std(~((predicted_labels - YTe) == 0));   % can also be calculated analytically from decoding_results using p(1-p) binomial variance formula
                                    curr_total_CV_zero_one_loss_results{iTrainingInterval} = [curr_total_CV_zero_one_loss_results{iTrainingInterval}; ~((predicted_labels - YTe) == 0)]; 
                                else
                                    curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.all_single_CV_vals(iCV, iTrainingInterval, iTestInterval) = std(~((predicted_labels - YTe) == 0));   % can also be calculated analytically from decoding_results using p(1-p) binomial variance formula
                                    curr_total_CV_zero_one_loss_results{iTrainingInterval,iTestInterval} = [curr_total_CV_zero_one_loss_results{iTrainingInterval,iTestInterval}; ~((predicted_labels - YTe) == 0)];
                                end
                                

                                curr_correct_class_decision_values = [];
                                curr_rank_results = [];
                                
                                
                                
                                if (cv.save_results.decision_values == 1) ||  (cv.save_results.normalized_rank == 1)
                                
                                    % whether one needs to get the rank confusion matrix information (using this variable can slightly speed up the code)
                                    get_confusion_matrix_info = 0;
                                    if (cv.confusion_matrix_params.create_confusion_matrix ~= 0)
                                        if (iTrainingInterval == iTestInterval) 
                                            get_confusion_matrix_info = 1;
                                        else
                                            if ~(cv.confusion_matrix_params.save_confusion_matrix_only_train_and_test_at_same_time == 1), get_confusion_matrix_info = 1; end    
                                        end
                                    end
                                        

                                    % get the normalized rank results, the decision values for the correct class, and rank confusion matrix information
                                    
                                    [curr_correct_class_decision_values curr_normalized_rank_results curr_rank_confusion_matrix] = cv.get_rank_and_decision_value_results(YTe, classifier.labels, decision_values, get_confusion_matrix_info);
                                    
                                    
                                    if cv.save_results.normalized_rank == 1
                                    
                                        if test_only_at_training_times == 1
                                            curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.decoding_results(iCV, iTrainingInterval) = mean(curr_normalized_rank_results);
                                            curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.all_single_CV_vals(iCV, iTrainingInterval) = std(curr_normalized_rank_results);
                                            curr_total_CV_normalized_rank_results{iTrainingInterval} = [curr_total_CV_normalized_rank_results{iTrainingInterval}; curr_normalized_rank_results'];                                             
                                        else
                                            curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.decoding_results(iCV, iTrainingInterval, iTestInterval) = mean(curr_normalized_rank_results);
                                            curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.all_single_CV_vals(iCV, iTrainingInterval, iTestInterval) = std(curr_normalized_rank_results);
                                            curr_total_CV_normalized_rank_results{iTrainingInterval,iTestInterval} = [curr_total_CV_normalized_rank_results{iTrainingInterval,iTestInterval}; curr_normalized_rank_results'];   
                                        end
                                        
                                        
                                        % save rank result stuff
                                        if cv.confusion_matrix_params.create_confusion_matrix ~= 0  


                                            YTe_unique_values = unique(YTe);
                                            
                                             % initialize all_rank_results_confusion_matrices
                                             if isempty(curr_bootstrap_rank_confusion_matrix)  %~exist('curr_bootstrap_rank_confusion_matrix')  
                                                 if  cv.confusion_matrix_params.save_confusion_matrix_only_train_and_test_at_same_time == 1
                                                     curr_bootstrap_rank_confusion_matrix = zeros(length(YTe_unique_values), length(YTe_unique_values), size(all_XTr, 2));
                                                 else
                                                     curr_bootstrap_rank_confusion_matrix = zeros(length(YTe_unique_values), length(YTe_unique_values), size(all_XTr, 2), size(all_XTe, 2));
                                                 end
                                             end


                                            % save rank confusion matrix stuff 
                                            if cv.confusion_matrix_params.save_confusion_matrix_only_train_and_test_at_same_time == 1
                                                if iTrainingInterval == iTestInterval
                                                    curr_bootstrap_rank_confusion_matrix(:, :, iTrainingInterval) = squeeze(curr_bootstrap_rank_confusion_matrix(:, :, iTrainingInterval)) + curr_rank_confusion_matrix;  % using this to get the average predicted ranking for each actual class
                                                end
                                            else
                                                curr_bootstrap_rank_confusion_matrix(:, :, iTrainingInterval, iTestInterval) = squeeze(curr_bootstrap_rank_confusion_matrix(:, :, iTrainingInterval, iTestInterval)) + curr_rank_confusion_matrix; %hopefully the orientation is right  % using this to get the average predicted ranking for each actual class
                                            end

                                        end

                                    end
                                       
                                    
                                    % save decision value information
                                    if cv.save_results.decision_values == 1
                                        
                                        if test_only_at_training_times == 1
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.decoding_results(iCV, iTrainingInterval) = mean(curr_correct_class_decision_values);  
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.stdev.all_single_CV_vals(iCV, iTrainingInterval) = std(curr_correct_class_decision_values);
                                            curr_total_CV_decision_values_results{iTrainingInterval} = [curr_total_CV_decision_values_results{iTrainingInterval}; curr_correct_class_decision_values'];        
                                        else
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.decoding_results(iCV, iTrainingInterval, iTestInterval) = mean(curr_correct_class_decision_values);  
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.stdev.all_single_CV_vals(iCV, iTrainingInterval, iTestInterval) = std(curr_correct_class_decision_values);
                                            curr_total_CV_decision_values_results{iTrainingInterval,iTestInterval} = [curr_total_CV_decision_values_results{iTrainingInterval,iTestInterval}; curr_correct_class_decision_values'];
                                        end
                                    end
                                    
                                    
                                    if cv.save_results.extended_decision_values == 1
                                        if test_only_at_training_times == 1
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.classifier_decision_values(iCV, :, iTrainingInterval) = curr_correct_class_decision_values;  % could just store the mean... hmmm...
                                        else
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.classifier_decision_values(iCV, :, iTrainingInterval, iTestInterval) = curr_correct_class_decision_values;  % could just store the mean... hmmm...
                                        end
                                    end
                                    
                                    % saving extended decision values for further analysis (this could take up a lot of memory/disk space) 
                                    if cv.save_results.extended_decision_values == 2
                                        if test_only_at_training_times == 1
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.all_classifier_decision_values(iCV, :, :, iTrainingInterval) = decision_values;  
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.all_classifier_decision_labels(iCV, :, iTrainingInterval) = YTe;                                        
                                        else
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.all_classifier_decision_values(iCV, :, :, iTrainingInterval, iTestInterval) = decision_values;  
                                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.all_classifier_decision_labels(iCV, :, iTrainingInterval, iTestInterval) = YTe;  
                                        end
                                    end

                                    
                                end  % end for saving rank results and decision values


                                

                                % save area under ROC curve results
                                 if (cv.save_results.ROC_AUC == 1) || (cv.save_results.ROC_AUC == 2) || (cv.save_results.ROC_AUC == 3)


                                    unique_labels = unique(YTe); 
                                    for iClass = 1:size(decision_values, 2)

                                        decision_vals_label_present = decision_values((classifier.labels(iClass) == YTe), iClass);
                                        decision_vals_label_absent = decision_values((classifier.labels(iClass) ~= YTe), iClass);

                                        if (cv.save_results.ROC_AUC == 1) || (cv.save_results.ROC_AUC == 3)  % if saving separate CV ROC results
                                            if test_only_at_training_times == 1
                                                curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results(iCV, iClass, iTrainingInterval) = get_AUC(decision_vals_label_present,  decision_vals_label_absent);
                                            else
                                                curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results(iCV, iClass, iTrainingInterval, iTestInterval) = get_AUC(decision_vals_label_present,  decision_vals_label_absent);
                                            end
                                        end
                                        
                                        
                                        if (cv.save_results.ROC_AUC == 1) || (cv.save_results.ROC_AUC == 2)  % if saving combined CV ROC results 
                                        
                                            if iCV == 1
                                                if test_only_at_training_times == 1
                                                    all_CV_decision_vals_label_present{iClass, iTrainingInterval} = []; 
                                                    all_CV_decision_vals_label_absent{iClass, iTrainingInterval}  = []; 
                                                else
                                                    all_CV_decision_vals_label_present{iClass, iTrainingInterval, iTestInterval} = []; 
                                                    all_CV_decision_vals_label_absent{iClass, iTrainingInterval, iTestInterval}  = []; 
                                                end
                                            end
                                            
                                            if test_only_at_training_times == 1
                                                all_CV_decision_vals_label_present{iClass, iTrainingInterval}  = [all_CV_decision_vals_label_present{iClass, iTrainingInterval};  decision_vals_label_present];
                                                all_CV_decision_vals_label_absent{iClass, iTrainingInterval}  = [all_CV_decision_vals_label_absent{iClass, iTrainingInterval};  decision_vals_label_absent]; 
                                            else
                                                all_CV_decision_vals_label_present{iClass, iTrainingInterval, iTestInterval}  = [all_CV_decision_vals_label_present{iClass, iTrainingInterval, iTestInterval};  decision_vals_label_present];
                                                all_CV_decision_vals_label_absent{iClass, iTrainingInterval, iTestInterval}  = [all_CV_decision_vals_label_absent{iClass, iTrainingInterval, iTestInterval};  decision_vals_label_absent]; 
                                            end
                                                
                                        end
                                        
                                    end


                                    % save stdev over classes
                                    if (cv.save_results.ROC_AUC == 1) || (cv.save_results.ROC_AUC == 3)
                                        if test_only_at_training_times == 1
                                            curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_classes(iCV, iTrainingInterval) = std(squeeze(curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results(iTrainingInterval, iCV, :)));
                                        else
                                            curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_classes(iCV, iTrainingInterval, iTestInterval) = std(squeeze(curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results(iTrainingInterval, iTestInterval, iCV, :)));
                                        end
                                    end
                                    
                                 end



                            end  % end for the test time periods

                            
                            
                    end  % end for the train time periods


                    
                end  % end for the CV blocks
                

% changed stuff but to here, but there seems to be a cleaner way to do things ...


                
                % save more measures of decoding result variability
                for iTrainingInterval = 1:size(all_XTr, 2)
                    
                    if cv.test_only_at_training_times == 1  
                        test_interval = iTrainingInterval; 
                    else 
                        test_interval = 1:size(all_XTe, 2); 
                    end
                    
                     
                    for iTestInterval = test_interval 
                        
                        curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.all_single_CV_vals_combined(iTrainingInterval, iTestInterval) = std(curr_total_CV_zero_one_loss_results{iTrainingInterval, iTestInterval});
                        curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.over_CVs(iTrainingInterval, iTestInterval) = std(squeeze(curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.decoding_results(iTrainingInterval, iTestInterval, :)));
               
 
                        
                         if save_normalized_rank == 1
                             curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.all_single_CV_vals_combined(iTrainingInterval, iTestInterval) = std(curr_total_CV_normalized_rank_results{iTrainingInterval, iTestInterval});
                             curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.over_CVs(iTrainingInterval, iTestInterval) = std(squeeze(curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.decoding_results(iTrainingInterval, iTestInterval, :)));
                         end
                                                   
                        if save_decision_values == 1 
                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.stdev.all_single_CV_vals_combined(iTrainingInterval, iTestInterval) = std(curr_total_CV_decision_values_results{iTrainingInterval, iTestInterval});
                            curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.stdev.over_CVs(iTrainingInterval, iTestInterval) = std(squeeze(curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.decoding_results(iTrainingInterval, iTestInterval, :)));
                        end
                       
                        
                        if (save_roc_auc == 1) || (save_roc_auc == 2) || (save_roc_auc == 3)
                            
                            if (save_roc_auc == 1) || (save_roc_auc == 2)
                                for iClass = 1:size(decision_values, 2)  
                                      curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.combined_CV_ROC_results.decoding_results(iTrainingInterval, iTestInterval, iClass) =  get_AUC(all_CV_decision_vals_label_present{iClass, iTrainingInterval, iTestInterval}, all_CV_decision_vals_label_absent{iClass, iTrainingInterval, iTestInterval});      
                                end  

                                % save combined stdev over classes
                                curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.combined_CV_ROC_results.stdev.over_classes(iTrainingInterval, iTestInterval)  = std(squeeze(curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.combined_CV_ROC_results.decoding_results(iTrainingInterval, iTestInterval, :)));

                            end
                                
                            % save separate stdevs
                            if (save_roc_auc == 1) || (save_roc_auc == 3)
                                curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_CVs(iTrainingInterval, iTestInterval) = std(squeeze(mean(curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results(iTrainingInterval, iTestInterval, :, :), 4)));
                            end
                            
                            % %DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_CVs(iTrainingInterval, iTestInterval, iBootstrap, iClass) = std(DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results(iTrainingInterval, iTestInterval, iBootstrap, :, iClass));
                            % %curr_combined_CV_and_class_data = squeeze(DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results(iTrainingInterval, iTestInterval, iBootstrap, :, :));
                            % %DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_CVs_and_classes_combined(iTrainingInterval, iTestInterval, iBootstrap) = std(curr_combined_CV_and_class_data(:));   % this is probably totally pointless
                            
                            
                        end
                        
                        
                    end
                    
                end
                
                
                
              
                % save some results for creating the rank confusion matrix
                % normalizing the rank confusion matrix so that each column (actual label) contains the real normalized rank values (i.e., values in the rang [0 1])
                if (cv.save_results.normalized_rank == 1) && (cv.confusion_matrix_params.create_confusion_matrix == 1) 

                    if cv.confusion_matrix_params.save_confusion_matrix_only_train_and_test_at_same_time == 1
                         for iCMTime = 1:size(all_XTr, 2)
                             total_num_test_examples_from_each_class = squeeze(sum(curr_bootstrap_rank_confusion_matrix(:, :, iCMTime)))./(length(unique(YTe))./2);   %  (length(unique(YTe))./2) is the sum of all the rank values, i.e., if k = 1/num_clases then this is 0 + 1/k + 2/k + ... + 1
                             curr_bootstrap_rank_confusion_matrix(:, :, iCMTime) = squeeze(curr_bootstrap_rank_confusion_matrix(:, :, iCMTime))./repmat(total_num_test_examples_from_each_class, length(unique(YTe)), 1);
                         end     
                         
                    else
                         for iCMTrain = 1:size(all_XTr, 2)
                             for iCMTest = 1:size(all_XTe, 2)   
                                total_num_test_examples_from_each_class = squeeze(sum(curr_bootstrap_rank_confusion_matrix(:, :, iCMTrain, iCMTest)))./(length(unique(YTe))./2);
                                curr_bootstrap_rank_confusion_matrix(:, :, iCMTrain, iCMTest) = squeeze(curr_bootstrap_rank_confusion_matrix(:, :, iCMTrain, iCMTest))./repmat(total_num_test_examples_from_each_class, length(unique(YTe)), 1);
                             end
                         end     
                    end

                     all_confusion_matrix_rank_results{iBootstrap} = curr_bootstrap_rank_confusion_matrix;   
                     curr_bootstrap_rank_confusion_matrix = [];


                end


                
                
                              
               % need to save things this way to have them work with parfor (matlab is lame)                 
               zero_one_loss_decoding_results(:, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.decoding_results;
               zero_one_loss_stdev_all_single_CV_vals(:, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.all_single_CV_vals;
               
               zero_one_loss_stdev_all_single_CV_vals_combined(:, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.all_single_CV_vals_combined;
               zero_one_loss_stdev_over_CVs(:, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.over_CVs;
               
               
               if save_normalized_rank == 1
                    normalized_rank_decoding_results(:, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.decoding_results;
                    normalized_rank_stdev_all_single_CV_vals(:, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.all_single_CV_vals;
                    normalized_rank_stdev_all_single_CV_vals_combined(:, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.all_single_CV_vals_combined;
                    normalized_rank_stdev_over_CVs(:, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.over_CVs;
               end
               
               % will need to expand this if saving all decision values...
               if save_decision_values == 1
                    decision_values_decoding_results(:, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.decoding_results;
                    decision_values_stdev_all_single_CV_vals(:, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.stdev.all_single_CV_vals;
                    decision_values_stdev_all_single_CV_vals_combined(:, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.stdev.all_single_CV_vals_combined;
                    decision_values_stdev_over_CVs(:, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.stdev.over_CVs;
               end
               
               if cv.save_results.extended_decision_values == 1
                    decision_values_classifier_decision_values(:, :, :, iBootstrap, :) = curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.classifier_decision_values;
               end

               if cv.save_results.extended_decision_values == 2
                    decision_values_all_classifier_decision_values(:, :, :, iBootstrap, :, :) = curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.all_classifier_decision_values;  
                    decision_values_all_classifier_decision_labels(:, :, :, iBootstrap, :) = curr_bootstrap_DECODING_RESULTS.DECISION_VALUES.all_classifier_decision_labels;  
               end
               

               if (save_roc_auc > 0) && (save_roc_auc < 4)
               
                   if (save_roc_auc == 1) || (save_roc_auc == 3)
                       roc_auc_separate_decoding_results(:, :, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results;
                       roc_auc_separate_stdev_over_classes(:, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_classes;      
                       roc_auc_separate_stdev_over_CVs(:, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_CVs;
                       
                   end
                   
                   if (save_roc_auc == 1) || (save_roc_auc == 2)
                       
                       roc_auc_combined_decoding_results(:, :, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.combined_CV_ROC_results.decoding_results;
                       roc_auc_combined_stdev_over_classes(:, :, iBootstrap) = curr_bootstrap_DECODING_RESULTS.ROC_AUC_RESULTS.combined_CV_ROC_results.stdev.over_classes;
                       
                   end
                       
               end
               

               if isfield(curr_bootstrap_DECODING_RESULTS, 'FP_INFO')
                    fp_info{iBootstrap} = curr_bootstrap_DECODING_RESULTS.FP_INFO;
               end
               
             
                % some additional information to save
                num_test_points_on_each_CV_run(iBootstrap) = length(YTe);   % assuming lenght(YTe) is always the same for all times (which I think has to be true)
                num_training_points_on_each_CV_run(iBootstrap) = length(YTr);
                dimension_of_data_points(iBootstrap) = size(XTr, 1);  
                the_unique_labels(iBootstrap, :) = unique(YTe);
                the_labels_YTe_one_split(iBootstrap, :) = YTe;
                num_unique_labels(iBootstrap) = length(unique(YTe));
                num_CV_runs(iBootstrap) = size(all_XTr{1}, 2);
                
                
 
                 % display progress of decoding procedure...
                  
                 if cv.display_progress.bootstrap_run_time  == 1
                     toc   % display how long a bootstrap iteration took
                 end
          

                 
               % display the progress of the results (made this a separate function to make code easier to read  - hopefully this will not slow things down too much passing DECODING_RESULTS as an argument)
                 if (cv.display_progress.zero_one_loss + cv.display_progress.normalized_rank + cv.display_progress.decision_values + cv.display_progress.separate_CV_ROC_results + cv.display_progress.combined_CV_ROC_results) > 0
                      cv.display_result_progress(curr_bootstrap_DECODING_RESULTS, cv.display_progress);
                 end   
                
               
                
             end  % end for the bootstrap data


            
             
             

            % need to save things back into DECODING_RESULTS to get things working with parfor...
            
            % save feature preprossing data (seems inefficient, but I can't find a better way to do this that will work the parallel toolbox)
            if exist('fp_info')
                for iBoot = 1:length(fp_info) 
                     for iFP = 1:size(fp_info{1}, 2)                     
                         if ~isempty(fp_info{iBoot}{iFP})

                             fp_data_curr_alg = fp_info{iBoot}{iFP};
                             field_names = fields(fp_data_curr_alg);  
                             for iFieldName = 1:length(field_names)

                                for iTrainTime = 1:size(fp_data_curr_alg, 1)
                                    for iCV = 1:size(fp_data_curr_alg, 2)

                                        eval(['curr_field_data = fp_data_curr_alg(iTrainTime, iCV).' field_names{iFieldName} ';']); 

                                        if isnumeric(curr_field_data)
                                            eval(['DECODING_RESULTS.FP_INFO{iFP}.'  field_names{iFieldName} '(iTrainTime, iCV, iBoot, :) = curr_field_data;']);    % will have to modify this slightly if returning a matrix, etc..
                                        else
                                            eval(['DECODING_RESULTS.FP_INFO{iFP}.'  field_names{iFieldName} '{iTrainTime, iCV, iBoot} = curr_field_data;']); 
                                        end  

                                    end
                                end

                            end

                         end
                     end     
                end
            end
            

            
            DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.decoding_results = zero_one_loss_decoding_results;
            DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.all_single_CV_vals = zero_one_loss_stdev_all_single_CV_vals;
            DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.all_single_CV_vals_combined = zero_one_loss_stdev_all_single_CV_vals_combined;                               
            DECODING_RESULTS.ZERO_ONE_LOSS_RESULTS.stdev.over_CVs = zero_one_loss_stdev_over_CVs; 

            if cv.save_results.normalized_rank == 1
                DECODING_RESULTS.NORMALIZED_RANK_RESULTS.decoding_results = normalized_rank_decoding_results;
                DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.all_single_CV_vals = normalized_rank_stdev_all_single_CV_vals;
                DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.all_single_CV_vals_combined = normalized_rank_stdev_all_single_CV_vals_combined;
                DECODING_RESULTS.NORMALIZED_RANK_RESULTS.stdev.over_CVs = normalized_rank_stdev_over_CVs;   
            end

            if cv.save_results.decision_values == 1
                DECODING_RESULTS.DECISION_VALUES.decoding_results = decision_values_decoding_results;
                DECODING_RESULTS.DECISION_VALUES.stdev.all_single_CV_vals = decision_values_stdev_all_single_CV_vals;
                DECODING_RESULTS.DECISION_VALUES.stdev.all_single_CV_vals_combined = decision_values_stdev_all_single_CV_vals_combined;
                DECODING_RESULTS.DECISION_VALUES.stdev.over_CVs = decision_values_stdev_over_CVs;
            end

            if cv.save_results.extended_decision_values == 1
                DECODING_RESULTS.DECISION_VALUES.classifier_decision_values = decision_values_classifier_decision_values; 
            end

            if cv.save_results.extended_decision_values == 2
                DECODING_RESULTS.DECISION_VALUES.all_classifier_decision_values = decision_values_all_classifier_decision_values;  
                DECODING_RESULTS.DECISION_VALUES.all_classifier_decision_labels = decision_values_all_classifier_decision_labels;  
            end
            
            if (cv.save_results.ROC_AUC == 1) || (cv.save_results.ROC_AUC == 2)
                DECODING_RESULTS.ROC_AUC_RESULTS.combined_CV_ROC_results.decoding_results = roc_auc_combined_decoding_results;
                DECODING_RESULTS.ROC_AUC_RESULTS.combined_CV_ROC_results.stdev.over_classes = roc_auc_combined_stdev_over_classes;
            end

            if (cv.save_results.ROC_AUC == 1) || (cv.save_results.ROC_AUC == 3)
                DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.decoding_results = roc_auc_separate_decoding_results;                                                 
                DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_CVs = roc_auc_separate_stdev_over_CVs;                                                
                DECODING_RESULTS.ROC_AUC_RESULTS.separate_CV_ROC_results.stdev.over_classes = roc_auc_separate_stdev_over_classes;                    
            end


            

            % save additional results (code have this code in the body of this method, but trying to make things more readable)
            DECODING_RESULTS = cv.save_more_decoding_measures(DECODING_RESULTS);


            % save additional parameters about the decoding experiment
            DECODING_RESULTS.BOOTSTRAP_CV_PARAMETERS.num_test_points_on_each_CV_run = num_test_points_on_each_CV_run(1);    % assuming lenght(YTe) is always the same for all times (which I think has to be true)
            DECODING_RESULTS.BOOTSTRAP_CV_PARAMETERS.num_training_points_on_each_CV_run = num_training_points_on_each_CV_run(1);
            DECODING_RESULTS.BOOTSTRAP_CV_PARAMETERS.dimension_of_data_points = dimension_of_data_points(1); 
            DECODING_RESULTS.BOOTSTRAP_CV_PARAMETERS.unique_labels = the_unique_labels(1, :);
            DECODING_RESULTS.BOOTSTRAP_CV_PARAMETERS.num_unique_labels = num_unique_labels(1);
            DECODING_RESULTS.BOOTSTRAP_CV_PARAMETERS.num_CV_runs = num_CV_runs(1);

            DECODING_RESULTS.BOOTSTRAP_CV_PARAMETERS.toolbox_version_number = get_ndt_version;
            

            % create the confusion matrices                                        
            if (cv.confusion_matrix_params.create_confusion_matrix == 1) || (cv.confusion_matrix_params.create_all_test_points_separate_confusion_matrix == 1)

                % created a separate function for this.  This could make the code more memory intensive, but the code is now easier to read.
                % I am assuming YTe is the same for all time periods, boostrap iterations and CV trials, which should always be met as far as I can tell)
                % confusion matrix => predicted_label x real_label x num_time_periods x num_time_periods

               DECODING_RESULTS = create_confusion_matrices(cv, the_labels_YTe_one_split(1, :), all_predicted_labels, DECODING_RESULTS);

               if exist('all_confusion_matrix_rank_results')    % save rank confusion matrix
                   total_rank_cm = zeros(size(all_confusion_matrix_rank_results{1}));
                   for iBoot = 1:cv.num_bootstrap_runs
                       total_rank_cm = all_confusion_matrix_rank_results{iBoot}  + total_rank_cm;
                   end
                   DECODING_RESULTS.NORMALIZED_RANK_RESULTS.confusion_matrix_results.rank_confusion_matrix = total_rank_cm./cv.num_bootstrap_runs;
                   DECODING_RESULTS.NORMALIZED_RANK_RESULTS.confusion_matrix_results.rank_confusion_matrix_label_mapping = the_labels_YTe_one_split(1, :);
               end
               
            end

            
            
            

        end    % end run_cv_decoding
    
    

    end   % end methods 
    
    
    methods (Access = 'private')
    
        display_result_progress(cv, curr_results_to_display, display_progress);
        DECODING_RESULTS = create_confusion_matrices(cv, YTe, all_predicted_labels, DECODING_RESULTS);
        DECODING_RESULTS = save_more_decoding_measures(cv, DECODING_RESULTS);
        %FP_info_to_save = process_FP_info_to_save(cv, preprocessing_information_to_save);
        FP_info_to_save = process_FP_info_to_save(cv, preprocessing_information_to_save, iCV, iTrainingInterval, FP_info_to_save);
        [correct_class_decision_values normalized_rank_results rank_confusion_matrix] = get_rank_and_decision_value_results(cv, YTe, classifier_labels, decision_values, create_rank_confusion_matrix);

    end  % end private methods
        
    
end   % end class

